Prefill batching logic to handle chunked prefill/prefix caching for HPU #753
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hlin99
requested review from
adobrzyn,
afierka-intel,
iboiko-habana,
kamil-kaczor,
ksmusz,
kzawora-intel,
mgawarkiewicz-intel,
michalkuligowski and
xuechendi
as code owners
Due to HPU padding constraints, batching requests with existing history (ctx != 0) causes excessive memory usage, as the entire batch must be padded to the longest context, leading to OOM. This patch enforces a batch size of 1 for prefill operations when ctx != 0. Although this sacrifices some throughput in corenr cases, it effectively eliminates the OOM risk. Signed-off-by: Tony Lin <tony.lin@intel.com>
✅ CI PassedAll checks passed successfully against the following vllm commit: |
hlin99
changed the title
Collaborator
|
@hlin99 , hmm, after second thought, I am a little unsure how the changes impact unified attention. Will need @kzawora-intel to check |
Contributor
Author
sure. i wan't aware ua goes to the same code path. if ua can overcome HPU padding nature, we can definitely split the code into ua & non-ua paths. @kzawora-intel please advise. thx |
afierka-intel
approved these changes
Jan 13, 2026
jinyouzhi
pushed a commit
to jinyouzhi/vllm-gaudi
that referenced
this pull request
Jan 14, 2026
…PU (vllm-project#753) Logic to handle chunked prefill/prefix caching for HPU Due to HPU padding constraints, batching requests with existing history (ctx != 0) causes excessive memory usage, as the entire batch must be padded to the longest context, leading to OOM. This patch enforces a batch size of 1 for prefill operations when ctx != 0. Although this sacrifices some throughput in corenr cases, it effectively eliminates the OOM risk. Signed-off-by: Tony Lin <tony.lin@intel.com> Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
hlin99
added a commit
to hlin99/vllm-gaudi
that referenced
this pull request
Jan 26, 2026
…PU (vllm-project#753) Logic to handle chunked prefill/prefix caching for HPU Due to HPU padding constraints, batching requests with existing history (ctx != 0) causes excessive memory usage, as the entire batch must be padded to the longest context, leading to OOM. This patch enforces a batch size of 1 for prefill operations when ctx != 0. Although this sacrifices some throughput in corenr cases, it effectively eliminates the OOM risk. Signed-off-by: Tony Lin <tony.lin@intel.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Logic to handle chunked prefill/prefix caching for HPU
Due to HPU padding constraints, batching requests with existing
history (ctx != 0) causes excessive memory usage, as the entire
batch must be padded to the longest context, leading to OOM.
This patch enforces a batch size of 1 for prefill operations when
ctx != 0. Although this sacrifices some throughput in corenr cases,
it effectively eliminates the OOM risk.